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ABSTRACT 


Online planning of good teaching sequences has the poten- 
tial to provide a truly personalized teaching experience with 
a huge impact on the motivation and learning of students. In 
this work we compare two main approaches to achieve such 
a goal, POMDPs that can find an optimal long-term path, 
and Multi-armed bandits that optimize policies locally and 
greedily but that are computationally more efficient while 
requiring a simpler learner model. Even with the availabil- 
ity of data from several tutoring systems, it is never possible 
to have a highly accurate student model or one that is tuned 
for each particular student. We study what is the impact 
of the quality of the student model on the final results ob- 
tained with the two algorithms. Our hypothesis is that the 
higher flexibility of multi-armed bandits in terms of the com- 
plexity and precision of the student model will compensate 
for the lack of longer term planning featured in POMDPs. 
We present several simulated results showing the limits and 
robustness of each approach and a comparison of heteroge- 
neous populations of students. 


1. INTRODUCTION 


The current advances and ubiquity of learning and teaching 
technologies have the potential to improve education acces- 
sibility and personalization. Intelligent Tutoring Systems 
(ITS) have been proposed to make education more accessi- 
ble, more effective, and as a way to provide useful objective 
metrics on learning [1]. 


A major aspect of personalized education is to be able to 
identify the current level of students and how to address 
particular difficulties in the student learning process. The 
goal is to be able to choose online the activity that better ad- 
dresses the challenges being encountered by each particular 
student. Even two students with the same knowledge will 
require different activities to progress further due to their 
previous experience, cognitive skills or preferences. This is 
a difficult challenge because as ITS are encountering the 
students for the first time, it is difficult to know what is 


Proceedings of the 9th International Conference on Educational Data Mining 


Pierre-Yves Oudeyer 
Inria, France 
pierre-yves.oudeyer Ginria.fr 


Manuel Lopes 
Inria, France 
INESC-ID, Instituto Superior 
Técnico, Portugal 


manuel.lopesQinria.fr 


the impact of each activity on their progress. A commonly 
used method is to exploit a population-wide model on how 
students learn and assume that they are all similar. The 
personalization in such an approach is limited to adapting 
to student’s knowledge levels but assumes that the impact 
of each exercise is the same for all students with the same 
knowledge levels. 


Different methods have been proposed to handle this prob- 
lem. One popular and well-known method is the Partially 
Observable Markov Decision Process (POMDP) framework 
which has been proposed in different ways to select the op- 
timal activities to propose to a learner [13]. This frame- 
work can find the optimal teaching trajectories for a given 
teaching scenario model if an accurate student model is pro- 
vided which is not always possible in practice. The main 
drawback is the high computational complexity and as a 
consequence, only the simplest cases can be solved exactly. 
Another method explored recently to select optimized ac- 
tivities is to use the Multi-Arm Bandit (MAB) framework 
to personalize sequences of pedagogical activities [6]. These 
methods optimize learning in the short term (rather than 
in the long-term) and rely on much simpler student models 
while being computationally very efficient. 


In this paper, we compare the POMDP framework and the 
MAB framework (specifically the algorithm ZPDES already 
evaluated in real classrooms [6]). We first introduce a stu- 
dent model used to compare the different algorithms. We 
hen propose ways to model the heterogeneity in classrooms 
by considering that different students will have not only dif- 
ferent learning parameters but also that they might have 
different dependencies between the different knowledge com- 
ponents (KCs). Our experiments will evaluate how well a 
MAB can approach the optimal solution of a POMDP, and 
how the different algorithms behave when encountering a 
heterogeneous group of student. 


2. RELATED WORK 


In this work we are interested in the impact of the quality 
of the student models on the quality of the sequences of 
activities chosen by online algorithms. 


There are several approaches to automatically choose exer- 
cises based on the current knowledge level of students. We 
are here particularly interested in optimization methods that 
rely on minimal prior assumptions about the students or the 
knowledge domain. 
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One option already explored is the use of a partial-observable 
Markov decision process (POMDP) [13], [14]. PODMPs of- 
fer an appealing theoretical framework that guarantees an 
optimal long-term solution for a planning problem. How- 
ever, in general, as the computational complexity is high, 
it is practically impossible to find an exact solution to the 
problem. Some approximate solutions in the domain of ITS 
have considered the use of aggregations of states instead of 
tracking the full knowledge components. Another drawback 
is that POMDPs require a precise student model for which 
the policy is optimized. If the real student encountered de- 
viates from this model, then the optimality properties are 
lost. 


A more recent approach is to use the Multi-Arm Bandit 


have the advantage of being extremely computationally ef- 
ficient and rely on very weak student models. The main 
drawback is that there is no long-term planning of the best 
sequence of activities relying on an exploration-exploitation 
radeoff to find the best path. Aware of such problem, au- 
hors of one such algorithm considered that standard MAB 
needs to be complemented with a weakly specified knowl- 
edge graph to provide a long-term view on the optimization 
6]. 


As noted, before optimizing the sequence of exercises it is 
important to have some knowledge about the impact of a 
given exercise in the learning of the KCs, and also to be 
able to track what each student already masters. A large 
part of ITS research has been on the modeling aspects of 
the cognitive and student models. A seminal work on this 
topic was the Knowledge Tracing framework [7] which builds 
a detailed cognitive model of the student, of its learning pro- 
cesses by considering a set of independent KCs, the probabil- 
ity of learning them and the probability of correct or wrong 
answer in exercises that relies on those KCs. More recent 
methods extend this framework to a bayesian probabilis- 
tic approach [12, 15] improving the performance and under- 
standing of those methods. Recent methods have started to 
consider how to learn such models, and variants of it, allow- 
ing to simultaneously discover the relation between activities 
and KC, e.g. [8, 2, 5, 9]. 


As discussed these methods require an accurate knowledge 
of how students learn and require to track their mastery of 
each KC. For this, it is necessary to learn the constraints 
between different KC, exercises and KC. Given students’ 
particularities, it is impossible for a teacher to understand 
all the difficulties and strengths of individual students and 
provide an accurate student model manually. Even with 
the recent advances on model learning, there are several 
challenges in identifying parameters that best describe each 
individual student. These models have many parameters, 
and identifying all such parameters for a single student is a 
very hard problem due to the lack of data, often making the 
problem intractable. In most cases it is even impossible to 
identify some of the parameters [3, 4]. In the general case, 
it results in inaccurate models that cannot be exploited for 
individualized learning. Another problem is that these plan- 
ning methods are for a population of students and not for 
a particular student and this has already been proven to be 
suboptimal [11]. 
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MAB) framework to manage pedagogical activities [6]. MABs 


3. STUDENT MODELS 
3.1 Student model 


In this section, we will present the student model we will use, 
also called learner model in literature. We want a generative 
model that can simultaneously be used to predict students 
behaviour, model their knowledge acquisition and track their 
mastery level. For this, we built a student model, shown in 
Fig.1 similar to the Knowledge Tracing framework [10] and 
its variants. Similarly to [9], we include extra features in our 
model. We are particular interested in more realistic cases 
where each KC might depend on other KCs. In most cases 
it is assumed that each exercise just depends on one KC and 
that they are independent, this is not realistic most of the 
time, and such dependencies have a strong impact on the 
learning sequences generated by the different algorithms. 


Figure 1: Graphical model of the Student model, 
with L‘ the hidden state of the student at step n, 
a) activity proposed, and o”) the result obtained 
by the student. 


We consider a situation where a student has a set of m 
KCs K; to learn. A student’s state at step n is represented 
by the state of each KC, L™ = K™, dae K&, the global 
model is described on figure Fig.1. Each KC is defined by 
his state, mastered (K; = 1) or not mastered (K; = 0). 
For each KC, there is an initial probability of mastering 
it p( KO = 1) which is always null in our experiments to 
make students learn all the KCs through activities. The 
emission probabilities are defined by the guess probability, 
i.e performing correctly without mastering the skill, and the 
slip probability, i.e performing incorrectly despite mastering 
the knowledge. Theses probabilities are constant. Finally 
p(k” = 1|L@-),a™) defines the probability of transi- 
tion from not mastered to mastered K; while doing activity 
a at step n and depending of the constraints between KCs 
and their states. An activity can be represented as a vector 
@=Q1,..-,Q@m where a; = 1 if the activity allows to acquire 
Ki, a; = 0 else. The transition probability to learn a given 
KC K; at step n is given by the following formula: 


RL? =a a) Sabet Sea ky) A) 
j#i 
Where (;,; represent the probability to learn K; without 
considering other KCs and (;,; represent the impact of the 
KC K; on the probability to learn K;. If a given KC does 
not need other KCs to be learned, the term es Bij K; is 
null. 


For more simplicity, in our experiments, an activity a can 
provide an opportunity to acquire only one KC which in- 
duces an isomorphism between the knowledge space and the 
activity space. 
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3.2 Models of populations 

The previous model can be used to describe a single student 
or an average model of a population. Our goal is to under- 
stand the impact that the diversity of students has when the 
given sequence is optimized considering the same parameters 
for all students. We will achieve such goal by considering a 
canonical model and then make two types of disruptions: 
i) change the probabilities between the variables; ii) change 
the knowledge graph. 


The first way is to disrupt the parameters in the model, i.e. 
the probability of transition, guess, and slip. To do that, 
we consider that each parameter is sampled from a gaussian 
distribution. We can change the variance to increase the 
heterogeneity of the population. With a variance null, all the 
population has the same parameters. The second way is to 
change the knowledge graph that changes the dependencies 
between the different knowledge. This type of disruption can 
be small like adding or removing a dependency, or it can be 
as critical as rearranging completely the organization of the 
knowledge dependencies. These two types of disruption are 
combined in our experiments. 


LM: 0.2, Ky 0.2, Ko 0.2, Ks 0.2, Ka 0.2, ve 0.2, Ke 
BLO 7 0.1 0.2 0.2 0.2 
EMi: 65.  K3 —> Ky —> Ks —> Ko 
Bey Ko 0.1 
0.2, Ki 0.2, ve 0.2, Ki 0.2, Ke 
LM2: 42 0.2 
— K3 —> Ks 
LMs 0:2, Ky 0.2, K 0.2, a 0.2, Ks 0.2, Ki 0.2, Ke 
LMg 0:2, Ky 0.2, ve 0.2, Ka 0.2, Ks 0.2, Ks 0.2, Ke 


Figure 2: Knowledge graphs used in the simulations. 
ILMo is the nominal knowledge graph, with LM, 
and LM. introducing small disruptions in the pre- 
requirements between KCs. LM3 and LM, represent 
more critical disruptions that change the overall or- 
der of KCs. 


We used multiple knowledge graphs, shown in Fig.2. The 
arrows represent the dependencies between KCs. For ex- 
ample, [Mo represents a graph where the constraints be- 
tween the different KC are ordered in a linear way. Here, 
Bia = B21 = B32 = B43 = 65,4 = Bos = 0.2 and all the 
others values of §;,; are null. We then created several differ- 
ent transformations and variants to model different needs of 
the students in terms of the order of the different KC. 


LM, and LM: follow approximately the same overall se- 
quence of KC, but considering two initial branches for the 
different KC. LM, considers that KC, and KC» are inde- 
pendent and any of them allows to learn KC3. In these 
knowledge graphs, we can expect that optimizing for one 
will also work for the other as the overall sequence of KC is 
respected, even if the strategy is no longer optimal. We also 
created more critical disruptions in the knowledge graph. 
LMs3 and LM, present an inversion between two KCs. For 
LM3, KC, and KC; are inverted, what radically change the 
overall sequence of KCs. For 2Mg, it is K3 and K,4 that are 
inverted. 


Proceedings of the 9th International Conference on Educational Data Mining 


4. OPTIMIZING LEARNING POLICIES 


4.1 Partially Observed Markov Decision Pro- 
cess (POMDP) 


POMDP is a markovian decision process where the state is 
hidden and can only be inferred indirectly from the obser- 
vations. A POMDP consists of a tuple (S, A, Z,T, R,O,7) 
with S the state space, A the action space and Z the obser- 
vation space. T is the transition model, it gives the prob- 
abilities p(s’|s,a) of transitioning from state s to state s’ 
with the action a. O is the observation model, it gives the 
probabilities p(z|s,a) of having the observation z when ac- 
tion a is made in state s. R the cost model, it specifies the 
cost r(s,a) of choosing action a in state s, and the discount 
factor 7 gives the relation between immediate costs and de- 
layed costs. With all these components, the solution of a 
POMDP is a policy that optimizes total discounted future 
reward. 


This framework has been already used in the context of ITS 
[13]. The learner’s mastery is the hidden state s, learn- 
ing is the transition between states, the probabilities that 
the learner gives a good answer are given by the obser- 
vation model of the observation {correct, incorrect}. We 
use Perseus [14] as solver to find the optimal policy for our 
POMDP problem. 


4.2 Zone of Proximal Development and Em- 
pirical Success (ZPDES) 


Oo Active @ Deactivated @& Activity not Zone of Proximal 


_—‘ activity activity — explored Development __ 
8zpp > Azpo 
8zpp = Azpo Gm > Aa 8zpp > Azpo 8n2 2 Aa 


Time 


Figure 3: ZPDES exploration of an activity graph, 
with dzpp the success rate over all active activities, 
Azpp the threshold to expand the ZPD, 6,4; the suc- 
cess rate for the activity Az, and \, the threshold to 
reach to deactivate an activity. 


Here we present the recently introduced algorithm Zone of 
Proximal Development and Empirical Success (ZPDES) that 
is based on multi-armed bandits [6]. The idea of the algo- 
rithm is presented in Fig.3 and summarized in Alg.1. The 
algorithm follows an activity graph but goes through it in 
a stochastic way. ZPDES is initialized with a certain num- 
ber of activities defined as starting activities. At each point 
in time, ZPDES has a set of activities, called the zone of 
proximal development, that can be proposed to the student 
which is adapted depending on student result. In the ex- 
periments presented here, we make small changes in the ac- 
tivation/deactivation mechanism of the original algorithm. 
When the recent student success rate over all active activi- 
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ties dzpp reaches a value \z pp, the graph is expanded to ex- 
plore another activity and when the recent success rate for a 
particular activity 62, is higher than a threshold Xq, this ac- 
tivity can be removed from the active list. This two thresh- 
old allow to partially configure the exploration behaviour of 
the algorithm. Inside the set of active activities, ZPDES pro- 
poses exercises proportionally to the recent learning progress 
obtained by that activity. The activity graph following the 
same structure than the knowledge graph, we can directly 
configure ZPDES with the same knowledge graph used to 
configure POMDP. 


Algorithm 1 ZPDES algorithm 


Require: Set of na activities A 
Require: ¢ rate of exploration 
Require: distribution for parameter exploration €, 
1: Initialize of quality wa uniformly 

2: while learning do 

3: Initialize ZPD 


4: {Generate exercise: } 

5 for a€ ZPD do 

6: Ue = = a : 

7 Pa = Wa(l—¢) + CEu 

8 Sample a proportional to pa 

9: end for 

10: Propose activity a 

11: Get student answer C; and compute reward: 


12:0 r= ea ah x oe ae zits 

13: wa + Bwat nr {Update quality of activity} 

14: Update ZPD based on activity graph and success rates 
15: end while 


5. EXPERIMENTS 


The goal of our experiments is to compare the impact of the 
knowledge about the students on the online algorithms for 
choosing exercises, namely POMDP and ZPDES. We will 
proceed to change the heterogeneity of the student popula- 
tions and see how much disruption each algorithm is able 
to adapt. Our comparative measure of performance is the 
average skill level overall knowledge and over time, for all 
the students in the population. 


We will compare the results obtained with two algorithms: 
POMDP and ZPDES. Each algorithm will have different 
variants based on the knowledge included on each of them. 
POMDP relies on a knowledge graph and the parameters 
of such graph. Each variant of POM DP, is characterized 
by a specific student model used to find the optimal policy. 
ZPDES has as information the knowledge graph, and some 
parameters describing how to traverse this graph, no partic- 
ular assumption is made about the probabilities of knowl- 
edge acquisition. ZPDES# is a variant of ZPDES with the 
corresponding graph x and using the parameters that were 
used in an other experiment in a real world situation [6] 
mostly hand-tuned with the help of a pedagogical expert. 
ZPDES; will also use the graph x but the parameters to 
traverse the graph are optimized for that particular graph 
using a greed search. During the optimization, we saw that 
the majority of parameters present average results and only 
extreme parameters gave critical results. 


Single model results. The first experiment will do a san- 
ity check to evaluate each algorithm in conditions where 
each student is the same in the population and each algo- 
rithm is configured for this model of student. We expect 
POMDP to have the best results and we want to see how 
far ZPDES will be from the optimal solution. A Random 
strategy which selects one activity randomly among all pos- 
sible is also presented in this first experiment to see the gain 
of the algorithms. 


Average KC level 


POMDP_O ZpdesH_O Zpdes*_0 
— Random 


“0 20 40 60 80 100 
Time 


Figure 4: Evolution of the average skill level for 600 
students modeled with LM 0 which activity are man- 
aged by POMDP, ZPDES*, ZPDES? configured for 
LM 0. Shaded area represents the standard error of 
the mean. 


Fig.4 shows the comparison of POMDP, ZPDES*, ZPDES” 
and Random with a population of 600 students modelled 
with the knowledge graphs LM 0. We can see POMDP is 
the best for all the models, closely followed by ZPDES*. 
ZPDES” give a slower learning than the two others. Un- 
surprisingly, for one particular model, POMDP has the best 
performance. The optimized ZPDES is very close in per- 
formance to POMDP. The results are similar for models 
1, 2, 3 and 4, the curves are not presented here for space 
reason. We can thus verify that the combination of knowl- 
edge graphs and the activity exploration rules provides a 
space of policies that is close to the optimal POMDP one. 
ZPDES? present the slowest population learning among the 
algorithms but as its configuration was not optimized for any 
particular model we can expect such result. 


These results show that the algorithms behave as expected 
and that ZPDES has the potential to be close to the optimal 
POMDP solution. 


Multi model results. We will now present the main results 
of this work with the comparison between POMDP, ZPDES* 
and ZPDES” when confronted with heterogeneous popula- 
tions of students. The protocol of the experiments is as 
follows. First we provide each algorithm with the informa- 
tion about a specific population of students and then we 
test the capability of the algorithms to address a different 
and diverse population of students. As described earlier, 
each algorithm is given information about a particular stu- 
dent model x, POMDP,; receives the graph and the student 
model parameters, ZPDES? receives the graph and explo- 
ration parameters optimized for that same graph, ZPDES! 
receives the graph and standard parameters for the graph 
exploration. We test different versions of each algorithm 
with a population composed of students following 3 differ- 
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Table 1: Performance position of each algorithm 
configuration for each setup. The rank of each al- 
gorithm configuration, and the average rank of each 
algorithm is presented for steps 50 and 200. 


Students 0,1,2 / Alg config 0,1,2 


Rank t 50 Rank t 200 
Algorithm | Per conf | Average | Per conf | Average 
POMDPo 1 1 
POMDP, 3 1 2 2 
POMDP2 4 3 
ZPDES? 3 1 
ZPDES? 3 3 1 1 
ZPDES3, 6 3 
ZPDES§ 2 1 
ZPDES}{ 3 2 2 2 
ZPDES3 5 3 
Students 0,3,4 / Alg config 0,3,4 
Rank t 50 Rank t 200 
Algorithm | Per conf | Average | Per conf | Average 
POMDPo 1 2 
POMDP3 2 1 3 2 
POMDP, 4 5 
ZPDES? 2 1 
ZPDES? 3 2 2 1 
ZPDESZ 4 3 
ZPDES§ 2 2 
ZPDES3 3 2 4 2 
ZPDES}4 4 4 
Students 2,3,4 / Alg config 0,1 
Rank t 50 Rank t 200 
Algorithm | Per conf | Average | Per conf | Average 
POMDPo 1 1 3 9 
POMDP, 4 6 
ZPDES# 2 1 
O 1 1 
ZPDES+ 3 2 
ZPDES§ 2 1 4 9 
ZPDES} 3 5 


ent knowledge graphs. The probabilistic parameters of the 
student models in the population follow a gaussian distri- 
bution. There is 200 students per graphs for a total of 600 
students. 


On figure 5 we can see the evolution of the average mastery 
level for all KCs. The table 1 presents the ranking of each 
version of the algorithms and the average ranking of each 
algorithm at step 50 and 200 according to the curves com- 
parison for each setup DMo,1,2, LMo,3,4, and LMo2.3,4. The 
table 2 presents the statistical significance tests at step 50 
and 200 for each setup and what is the best methods if the 
results are statistically significant. 


By comparing the different p-values, we can see that the 
differences between POMDP and ZPDES* are never signif- 
icant, but it’s not the case for ZPDES”. For the models 
LMo,1,2, at step 50, ZPDES” drops behind the two others, 
but it catches up rapidly with the two others and present the 
same results at step 200. So for models which are close to 
each other, the 3 algorithms present almost the same result. 
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Figure 5: Evolution of the average skill level for 
600 students with POMDP, ZPDES*, ZPDES”. For 
each curve, the number attached to the algorithm’s 
name indicate what knowledge graph has been used 
to configure the algorithm. Each curve shows the av- 
erage KC level of the student population over time 
for each algorithm configuration. In general ZPDES 
have better results than POMDP. Shaded area rep- 
resents the standard error of the mean. 
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Table 2: ANOVA p-values for each setup to ver- 
ify if the differences in the KC level distribution 
according to each algorithm are statistically signif- 
icant with the best algorithms in parenthesis when 
it is significant. We note P for POMDP and Z for 
ZPDES 


P/Z* P/ZU Z* (27 

LM | +t50|+t200| t 50 | t 200 | t 50 | t 200 
0.1.2 075 | .95 ]10°°] .82 |] .003 |] .87 
a (@) | |) 
24 | .90 17 | 107 89 | 107 
0,3,4 (Z") (Z") 
31 | .30 1s |10-°> | .77 | 10°” 
2,3,4 (Z") (Z") 


For the models [Mo,3,4, observations are different. At step 
50, all the algorithms seem to have approximately the same 
performance, even if ZPDES” seems a bit behind but it’s 
not significant (p-values at 0.17 and 0.89). But with time, 
it takes the lead and achieves the best performance at 200 
steps. So when there are two models critically different 
from another, ZPDES” presents the best results. For the 
last case, the population is constituted of students following 
LMz2,3,4 models, and the algorithms are configured for mod- 
els LMo,1. As for the previous case there is no differences at 
step 50 but ZPDES” presents the best results at step 200. 


ZPDES” provides the best result because its exploration 
parameters were not optimized for any particular knowledge 
graph, giving it higher adaptability and less constrains in 
the exploration. For a particular type of student model it 
will present worse performance than POMDP or ZPDES"*, 
but for a heterogeneous population, ZPDES”, being more 
adaptable, has the best performance. 


6. CONCLUSION 


In this work we considered student models where the knowl- 
edge components can have constraints among each other, 
allowing to model some kind of pre-requisites. Under dif- 
ferent student models we can find an optimal teaching se- 
quence using POMDP. Another alternative is the use of the 
recently proposed method ZPDES that is computationally 
more efficient but without optimality guarantees. Our goal 
was to test how robust each of these methods is in relation 
with ill-estimated parameters of the models, or even wrongly 
estimated relations between KCs. This corresponds to the 
more realistic case of heterogeneous classes of students. 


We showed that for the trivial situation where the students 
are perfectly modeled with the student model, ZPDES can 
achieve the same performance as the POMDP. For heteroge- 
neous populations again ZPDES can achieve solutions simi- 
lar to POMDP. The best algorithm was using ZPDES that 
uses parameters that are not optimized for no population in 
particular. By having more flexibility in the exploration it 
becomes more robust to changes in the population. 


We conclude that multi-armed bandits, when combined with 
an activity graph, are a best choice in comparison with 
POMDPs due to its computational efficiency and reliance 
on simpler student models. 
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The code to generate the graphics and the results is available 
at: github.com/flowersteam/kidlearn/tree/edm2016, 
follow the README. 
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